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Abstract 

In this paper, we study the restrictions of the complementary series representation onto 
a symplectic subgroup no bigger than half of the size of the original symplectic group. 

1 Introduction 

Let Sp(n,M.) be the universal covering of Sp(n, R). Sp(n,W) is a central extension of 
Sp{n,R): 

1 -4 C -> Sp(n,R) -4 5p(n,R) -> 1, 

where C = Z. The unitary dual of C is parametrized by a torus T. For each k £ T^jienotc 
the corresponding unitary character of C by \ K ■ We say that a representation 7r of Sp(n, R) 
is of class k if 7r|c = X K - Since C commutes with Spin, R), for any irreducible representation 
7r of 5p(n,R), 7r|c = X K for some «. 

Denote the projection Sp(n,R) — ► 5p(n,R) by p. For any subgroup 7? of Sp(n,M.), de- 
note the full inverse image p~ 1 (H) by H. We adopt the notation from [10] , Let P be the 
Siegel parabolic subgroup of Sp(n, R). One dimensional characters of P can be parametrized 
by (e, t) where e £ T and f 6 C. Let I(e,t) be the representation of Sp(n, R) induced from 
the one dimensional character parametrized by (e, f ) of P. If £ is purely imaginary, /(e, f) is 
unitary and irreducible. If t is real, then I(e, t) has an invariant Hermitian form. Sahi gives 
a classification of all irreducible unitarizable I(e,t). Besides the unitary principal series, 
there are complementary series C(e,t) for t in a suitable interval ( [10 ). 

Let (Sp(p, R), Sp(n—p, R)) be a pair of symplectic groups diagonally embedded in Sp(n, R). 
Suppose that p < n—p. Let U(n) be a maximal compact subgroup such that Sp{n—p, R) n 
J7(n) is a maximal compact subgroup of Sp(n — p, R). Denote — p, R) n t/(n) by 

fj(n-p). 

Theorem 1.1. Suppose that p <n—p and C(e,t) is unitary. Then 



p = [t|] is the best possible value for such a statement. In particular, for Sp(2m + 1,M) 

^( £ ' ^)lsp(m+l,H) ^ ^( e >*)lsp(TO+l,E)- 

To see this, let L 2 (Sp(n, K)) K be the set of functions with 

= M K M/(5) (* G C, 5 G Sp(n,R)); 
ll/ll 2 = / |/(ff)| 2 <%] (g 6 Sp(n,R), [5] e Sp(n,K)). 

JSp(n,M) 

We say that a representation of class k is tempered if it is weakly contained in L 2 (Sp(n, R)) K . 
By studying the leading exponents of I(e, 0) and C(e, t), it can be shown that I(e, 0)\^ rn+1 R ) 
is "tempered " and C(e, i)lg^( m +i r) i s not "tempered". Therefore 

/ ( £ '°)ls^(m+l,K) ^ C '( e '*)ls^(m+1,R)- 

The author would like to thank G. Olafsson and J. Lawson for some very help discussions. 



2 A Lemma on Friedrichs Extension 

Let S be a semibounded densely defined symmetric operator on a Hilbert space H . Suppose 
that (Su,u) > for every nonzero u <E T>(S). We call S positive. For u,v G T>(S), define 

(u,v) s = (u,Sv), 

||u||,s = (u,Su). 
Let Hs be the completion of 2? (5) under the norm || \\s- 

The operator S + I has a unique Friedrichs extension (S + I)o such that 2?((5 + /)o) C Hs+i 
and (u, w)s+j = (u, (S 1 + /)o^) for ah u G #s+/ and f G 2?((5 + I)q) (see Theorem in Page 
335 [8]). Here -ffs+j C H and (5 + i) is self-adjoint. Now consider {S + I)o — L It is an 
self-adjoint extension of S. It is nonnegative. By the spectral decomposition and functional 
calculus, (S + I)o — I has unique square root T (See 127. 128. [8]). 

Lemma 2.1. Let S be a positive densely defined symmetric operator. Then the square root 
of (S + I)o — I extends to an isometry from H$ into H . 

Proof: Clearly, the spectrum of T is contained in the nonnegative part of the real line. 
By spectral decomposition V((S + J) - I) = T>((S + I) ) C V{T) and TT = {S + I) - I. 
In addition for any u, v G T>(S) C D((5 + J) ), 

(Tu, Tv) = (u, TTv) = (u, (S + I) v - v) = (u, Sv) = (u, v)s- 

So T is an isometry from "D(S) into H . Since T>{S) is dense in Hs, T extends to an isometry 
from Hs into H. □ 

We denote the isometry by Is- It is canonical. 
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3 Complementary Series of Sp(n, R) 

Fix the Lie algebra sp(n, R): 

and the Siegel parabolic algebra p: 

<(o -x.) |y ' = y> - 

Fix the Levi decomposition p = [ © n with 

(={( o °_ x t ) \ X€ S l(n,R)} 

and 

Fix a Cartan subalgebra 

a = {diagOffi, H 2 , H n , -H u -H 2 ..., -H n ) \ H, e R}. 

Let /Sp(n, R) be the symplcctic group and P be the Siegel parabolic subgroup. Let LN be 
the Levi decomposition and A be the analytic group generated by the Lie algebra a. Clearly, 
L = GL(n,R) and L (~l U{n) = 0(n). On the covering group, we have L n C/(n) = O(n). 
Recall that 

U{n) = {(x,g) \ g £ U(n), exp 2nix = detg, x £ R}. 

Therefore 

0(n) = {(x, g) | g £ 0(n),exp2nix = detg,x £ R}. 
Notice that x £ \7L since det g = ±1. We have the following exact sequence 

1 -> SO{n) -> O(n) -> -> 1. 

Consequently, we have 

1 — > GL (n,R) -> Z -> ->• 1. 

In fact, 

£ = {(z, 5) I .9 G i,exp27ria; = j^^ ^ e M l- 

The one dimensional unitary characters of ^Z are parametrized by the one dimensional 
torus T. Identify T with [0, 1). Let /i c be the character of \7L corresponding to e £ [0, 1) 
Now each character yields a character of L, which in turn, yields a character of P. For 
simplicity, we retain [f to denote the character on L and P. Let v be the det-character on 
L , i.e., 

v{x,g) = \detg\ (x,g)£L. 
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Let _ 

I{e,t)=Indf {n ' W ' ) ^®v t 

be the normalized induced representation with e G [0, 1) and t G C. This is Sahi's notation 
applied to the universal covering of the symplectic group [10] , I(e,t) is a degenerate prin- 
cipal series representation. Clearly, I(e, t) is unitary when t is purely imaginary. 

When t is real and I{e, t) is unitarizable, the unitary representation, often denoted by C(e, t), 
is called a complementary series representation. Various complementary series of Sp(n, R) 
and its metaplectic covering was determined explicitly or implicitly by Kudla-Rallis, 0rsted- 
Zhang, Brason-Olafsson-0rsted and others. See [BJ, [2J, [7] and the references therein. 
The complete classification of the complementary series of the universal covering is due to 
Sahi ( Q]]]). 

Theorem 3.1 (Thm A, [10] )■ Suppose that t is real. For n even, I(e,t) is irreducible and 
unitarizable if and only if < \t\ < |4 — |2e — 1||. For n odd and n > 1, I(e,t) is irreducible 
and unitarizable if and only if < \t\ < ^ — |4 — |2e — 1||. 

Figure 1: Complementary Parameters (E, t) 

One can easily check that the complementary series exist if e ^ 0, \ for n odd and n > 1 ; 
if e 7^ j, | for n even. It is interesting to note that complementary series always exist unless 
I(e, t) descends into a representation of the metaplectic group. For the metaplectic group 
Mp(2n+1, R), there are two complementary series J(i,i)(0 < t < i) and l{j,t)(0 <t<\). 
For the metaplectic group Mp(2n, R), there are two complementary series 1(0, t)(0 < t < ^) 
and l(^,t)(0 < t < |). These four complementary series are the "longest ". 

For n = 1, the situation is quite different. The difference was pointed out in [B]. For 
example, there are Bargmann's complementary series representation for l(0,t)(t G (0, ^))- 
The classification of the complementary series of Sp(l, R) can be found in [I], [9], [5]. 

Since our restriction theorem only makes sense for n > 2. We will assume n > 2 from 
now on. The parameters for the complementary series of Sp(n, R) are illustrated in fig. [TJ 

4 The generalized compact model and The Intertwining 
Operator 

Recall that 

I°°(e, *) = {/€ C^iSpin, R)) | f(gln) = ® ^(P 1 )/^ G L, n G N)} 

where p = Let X = 5p(n,R)/P. Then J°°(e, t) consists of smooth sections of the 

homogeneous vector bundle £ £j t 

lSp(7l, R) Xp C^e^t + p -> X. 
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Since X = Sp(n,M)/P = U(n)/0(n), U(n) acts transitively on X. f G I co (e,t) is uniquely 
determined by f\jj^ and vice versa. Moreover, the homogeneous vector bundle £ €; t can be 
identified with JC t . t 

U(n) Xo(n) ^V<»^ t+p lo(ri) ~* % 

naturally. Notice that the homogeneuous vector bundle /C £i t does not depend on the pa- 
rameter t. We denote it by IC e . The representation I(e,t) can then be modeled on smooth 
sections of IC e . This model will be called the generalized compact model. 

The generalized compact model provides much convenience. First, it equips the smooth 
sections of /C £it with a pre-Hilbcrt structure 

(/i,/a)x= / fi(k)Mkjd[k}, 
J[k]ex 

where k G U(n) and [k] G X. It is easy to verify that /i(fc)/2(fc) is a function of [fc] and it 
does not depend on any particular choice of k. Notice that our situation is different from the 
compact model since U(n) is not compact. We denote the completion of I°° with respect to 
(, )x by Ix(e,t). Secondly, the action of U(n) on /C e induces an orthogonal decomposition 
of/x(e,t): 

Ix[e, t) = ® a e2^V{a + e(2, 2, . . . , 2)), 

where V(a + e(2, 2, . . . , 2)) is an irreducible representation of U(n) with highest weight 
a + e(2,2,...,2). Let 

V(e,t) = © ae2Z nV(a + e(2,2,...,2)). 

V(e, t) possesses an action of the Lie algebra sp(n, R). It is called the Harish- Chandra mod- 
ule of I(e,t). Clearly, V(e,t) C I°°(e,t) C I x (e,t). 

For each t, there is an Sp(n, R)-invariant sesquilinear pairing of /(e, t) and I(e, —t), namely, 

(h,h)= I h(k)h{k~)d[k], 

Jx 

where /i G I(t,t) and /a G /(e, — t). If i is purely imaginary, we obtain a Sp(n, R)-invariant 
Hermitian form which is exactly (, )x- Since (, )x is positive definite, I(e, t) is unitary. 

For each real t, the form (, ) gives an Sp(n, R)-invariant sesquilinear pairing of I(t,t) and 
I(e, —t). There is an intertwining operator 

A(e,t):V(e,t)^V(e,-t) 

which preserves the action of sp(n,R) (see for example [2]). Define a Hermitian structure 
(,) M onV(e,t) by 

(it, v) e ,t = (A(e, t)u, v), (u, v G V(e, t)). 
Clearly, (, ) <L:t is sp(n, R)-invariant. So A(e, t) induces an invariant Hermitian form on V(e, t). 

Now A(e,t) can also be realized as an unbounded operator on Ix(e,t) as follows. For 
each / G V(e,t), define Ax(e,t)f to be the unique section of C e ,t such that 

(Ax(e,t)f)\ 0{n) = (A(e,t)f)\ lJ{n) . 
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Notice that Ax(e,t)f G I(e,t) and A(e,t)f G I(e, -t). They differ by a multiplier. 

Now Ax(e, t) is an unbounded operator on the Hilbert space Ix(t,t). The following fact is 
well-known in many different forms. I state it in a way that is convenient for later use. 

Lemma 4.1. Lett G R. I(e,t) is unitarizable if an only ifAx(e,t) extends to a self-adjoint 
operator on Ix(e,t) with spectrum on the nonnegative part of the real axis. 

The spectrum of Ax(e,t) was computed in [2] and [7] explicitly for special cases 
and in [10] implicitly. In particular, Ax{e 7 t) restricted onto each [/(n)-type is a scalar 
multiplication and the scalar is bounded by a polynomial on the highest weight. We obtain 

Lemma 4.2. Ax(e,t) extends to an unbounded operator from I°°(e,t) to I°°(e,t). 

This lemma follows from a standard argument that the norm of each £/(n)-component 
in the Peter- Weyl expansion of any smooth section of IC e decays rapidly with respect to the 
highest weight. 

5 The Mixed Model 

Suppose that p + q = n and p < q. Fix a subgroup Sp(p, R) x Sp(q,R) in Sp(n,R). Then 
we have ajsubgroup Sp(p,R)Sp(q,R) in Sp(n,R). Notice that Sp(p,R) n Sp{q,R) = Z. So 
Sp(p,M)Sp(q, R) is not a direct product, but rather the product of the two groups as sets. 
Let U{q) = Sp(q,R) n U(n). 

Theorem 5.1 (Main Theorem). Suppose thatp + q = n andp < q. Given a complementary 
series representation C(e,t), 

In other words, there is an isometry between C(e,t) and I(e, 0) that intertwines the actions 
ofU{q) and ofSp{p,R). 

We begin by recall a result concerning the action of Sp(p, R) x Sp(q, R) on X ( [3]). 

Lemma 5.1. Sp(p,R) x Sp(q, R) acts on X with a unique open dense orbit Xq- Let p < q. 
Let P p ,2q-2p be a maximal parabolic subgroup of Sp(q, R) preserving an q — p dimensional 
isotropic subspace. Let GL q - p Sp(p,R)N Pt 2q~2p be the Langlands decomposition of Pp,2q-2p- 
Let 

H = {(u, m 2 t u^ 1 n 2 ) \ m 2 G GL(q — p,R), u G Sp(p, R), n 2 G N p , 2q-2p}- 

Then X = Sp(p,R) x Sp(q,R)/H. 

Notice that H = P p ,2q-2p- But the Sp(p,R) factor in P is diagonally embedded in 
Sp(p, R) x Sp(g,R). In particular, there is a principal fibration 

Sp(p,R) -^X ^ Sp(q,R)/P p ,2 q -2 P = U(q)/0(q-p)U(p). 

Here 0(q - p)U(p) = U{q) n J Pp, 2g -2p C Sp(q,R). Let M = 5p(p,M)C/(g) and M = 
Sp(p,R)U(q). Then £ £it restricted onto X becomes 

M X (^(p) C ^ M/0( 9 ^(p) £ -Xb. 
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Let I£°x ( € it) be the set of smooth section of £ e ^ that are compactly supported on Xq. 
Clearly 

J^ (e,t)c/»(^)- 

Consider the restriction of (, )x onto I£° x (e, t). We are interested in expressing (, )x as an 
integral on M/0(q — p)U(p). This boils down to a change of variables from U(n)/0(n) to 
M/0{q^p)U{p). 

Let dgi be the invariant measure on Sp(p, R) and d[k 2 ] be the invariant measure on U(q)/0(q — p)U(p). 
Every element in Sp(n, R) has a U(n)P$ decomposition where P is the identity com- 
ponent of P. For each g <E Sp(n, R), write g = u(g)p{g). For each gifc 2 G M, write 
<7ifc 2 = u{gik 2 )p{gik 2 ) . Then u defines a map from M to u induces a map from 

M/0(q — p)U(p) to U(n)/0(n) which will be denoted by j. Clearly, j is an injection from 

M/0(q - p)U{p) onto X . Change the variable on X from M/0(q - p)U{p) to U{n)/6{n). 
Let J([<?i], [fc 2 ]) be the Jacobian: 

flOgiUfa]) 

d[ffi]d[fe,] ' 

We have 

Lemma 5.2. Le£ 

A M ( 5 i,/e 2 ) = KM9ifc 2 )) t+ ' +2p J([.9i], [fe])- 
T/ien /or every /i, / 2 G ^°°(e, i) we /icwe 

(/i,/ 2 )x=/ /i(.9ifc 2 )/ 2 (5ifc 2 )A £ , t ( ffl ,fc 2 )d[ ffl ]d[fc 2 ] 

JM/0(q-p)U(p) 

where g x E Sp(p,R), k 2 G f/(g<), [.9i] G Sp(p, R) and [fc 2 ] G U(q)/0(q - p)U(p). Further- 
more, A Cit (gi, k 2 ) is a nonnegative right 0(q — p)U(p) -invariant function on Sp(p, R)xU(q). 
Proof: We compute 



/ . /i(5i fc 2)/2(ffife)A M (3i,fc 2 )d[3i]d[/c 2 ] 

JM/0{q-p)U(p) 

= [ _ /i(«(ffife))/ 2 (fi(gifc 2 ))i.(p(gifc2))-*" i - 2p Ae, t (5i,fc2)d[ffi]d[A :2 ] 

JM/0(a-v)U(v) 



lM/0(q-p)U(p) 

f. _ 

lM/0(q-p)U(p) 

: / _ /i(%fe))^W^M)" , " i "%(ft.fe)^ 1 (ji.*j)<i'([ji]W) 

JM/0(q-p)U(p) 



h{u)f 2 {u)d[u] = {h,h) x . 

(1) 

Since v{p{gik 2 ) and J([<7i], [fc 2 ]) remains the same when we multiply fc 2 on the right by 
0(q — p)U(p), A Ctt (gi, k 2 ) is a nonnegative right 0(q — p)[/(p)-invariant function. □ 

For each /i,/ 2 e /?° M (e,t), define 

{h,f2)M,t= / /i(5ifc2)/2(5ifc2)A £ . t (gifc 2 )d[ 5l ]c;[fc 2 ], 

JM 
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(/i,/2)m= / fi{gi,k 2 )f2{gik2)d[gi]d[k 2 ]. 
Jm 

Let lM(t,t) be the completion of I^M{e,t) under (, )u,t- Clearly 7^(e,i) = Ix(e,t) as 
Hilbert representations of Sp(n,M.). Jm(c>*) is called the mixed model. 

6 Mixed Model for Unitary Principal Series 

Lemma 6.1. If t is purely imaginary, then A e ^ t {gi, /c 2 ) is a constant and (, )m,t is a 
constant multiple of (, )m . 

Proof: Let t G iR. Let / x ,/ 2 G I°°(e,t) and ft, G Sp(p,R). We have 

{I{e,t)(h)h,I{e,t){h)f 2 ) x 
={I{e,t){h)hJ{e,t){h)h) M ,t 

= [ h{h- 1 gik 2 )J 2 lJF^^^ t {g ll k 2 )d[g 1 ]d[k2] (2) 

JM/0(q-p)U(p) 



fi(gik 2 ) f 2 {gi, k 2 )A ett (hgi, k 2 )d[gx}d[k 2 } 

M/0(q-p)U(p) 

Since J(e, t) is unitary, 

(I(e,t)(h)h,I(e,t)(h)f 2 ) x = (hJ 2 )x. 

We have 

/ . fi(gik 2 )f2(gik 2 )A e , t (hg 1 ,k 2 )d[g 1 }d[k 2 }= fi(gik 2 )f 2 (gik 2 )A eit (gi, k 2 )d[g 1 ]d[k 2 \ 

JM/0{q-p)U{p) JM/0(q-p)U{p) 

It follows that A e t (hgi, k 2 ) = A e j(gi, k 2 ). for any h G Sp(p, M). Similarly, we obtain 
A e ,t(<7i, fcfc 2 ) = A(c/i,fc 2 ) for any fe G U(q). Hence, A e ,t (31,^2) is a constant for purely 
imaginary i. □ 

Corollary 6.1. JQ51], [fc 2 ]) = cu{p{g 1 k 2 )y 2p and A e ,t(gi,k 2 ) = cv{p{g 1 k 2 )) t+t . Further- 
more, 

I M (e,t) S L 2 (M x Q( ^ (p) C^KKsi^'+MsiM^]). (3) 

From now on, identify L 2 (M x 0{q Ipju( p) C f Xp(5i fc 2)) t+t( %iMfe]) with I M (e,t). 

Corollary 6.2. v(p(gxk 2 ))~ p G L 2 (M fO(q — p)U{p)) and v{p{gik 2 ))~ 1 is a bounded pos- 
itive function. 

Proof: Since X is compact, J([gi], [fc 2 ]) G L 1 (M/0(g — p)U(p)). Notice that v(p(g 1 k 2 )y 2p = 

cJ([gi], [k 2 ]). It follows that v{p{ gi k2))- p e L 2 {M/0{q^p)U{p)). Since J([ 5 i], [fc 2 ]) is con- 
tinuous and [fc 2 ] is a compact manifold, it suffice to show that for each fc 2 , { J([<?i], [& 2 ])} gi eSp(p,R) 
is bounded. This is true because j\k 2 Sp(p.R) is an analytic compactification of Sp(p,M) and 
the Jacobian can be easily computed just like the way it is done in [3] (see Theorem 2.3). 
In particular, it is always positive. By Cor. 16.11 v(p(gik 2 ))~ 1 is bounded and positive. □ 

If / G I M (e,ti) and h > 0, we have WfWu.h > CWfWuM-h- So I M {e,h) C I M (e,h - h). 
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Corollary 6.3. Suppose that h > 0. Then /m(£,^i) C Im^^i — h) under the Equation [3J 

Notice that in the mixed model, the actions of Sp(j>, R) and U (q) does not depend on 
the parameter t. We obtain 

Theorem 6.1. Let t be purely imaginary. I(e,t) can all be modeled on 

In particular, ^M^^lg^pgjjj^-) — Im{^j ®")\sp(p it) (7(g) an< ^ ^ e identity operator intertwines 
/ M (e,0)|g- p(pjR)f7{g) with lM(e,t)\g- p{pR)0{q) . 

For i a nonzero real number, A € t (g, k) is not a constant. So Theorem 16. II does not hold 
for real nonzero t. 



7 "Square Root "of the Intertwining Operator 

Suppose from now on t £ R. For / £ Z°° (e, define a function on M, 

(A M (e,t)/)(<7iA; 2 ) = A{e,t)f{g x k 2 ) ( 9l £ Sp(p,R),fc 2 £ £%)). 

So j4ju(e,t) is the "restriction "of A(e,t) onto M . Aj^(e,t) is not yet an unbounded operator 
on Ij^{e,t). In fact, for t > 0, A^(e, i) does not behave well. In this case, it is not clear 
whether Aj^(e,t) can be realized as an unbounded operator on Ij^{e,t). AM(e,t)f differs 
from A x (e,t)f by a multiplier. 

Lemma 7.1. For t £ R and f £ I°°(e, t), 

{A M {e,t)f\fy){ gi k 2 ) = (A x (e,t)f)( gi k 2 n P ( gi k 2 )) 2t = (A x (e,t)f)( gi k 2 )A e . t ( gil k 2 ). 
This Lemma is due to the fact that A x (e, t)f £ J(e, t) but A(e, t)f £ 7(e, — i). 

Let / £ I°°(e,t). In terms of the mixed model, the invariant Hermitian form (, ) e ,t can 
be written as follows: 

(fJ)e,t = (A x (e,t)fJ) x = f A M (e,t)f\ Al 7\ £l d[ gi }d[k 2 }. 

JM/0(q-p)U(p) 

We obtain 

Lemma 7.2. For f u f 2 £ I°°(e,t), (/i,/ 2 ) M = (^(e, t)/i| A , / 2 |m)m- 

Theorem 7.1. Ift<0 and C(e, i) is a complementary series representation, then A]\i(e, t) 
is positive and densely defined symmetric operator. Its self-adjoint-extension (^(e, i) + 
J)o — / /las a unique square root which extends to an isometry from C(e, i) onto 
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Proof: Let t < 0. Put 

- 2/ 



Let / 6 I°°(e,t). Then 



A M (e,t)(f\u) = v(p(9ik2)) 2t A x (e,t)f( gi k 2 ). 



By Lemma 17.11 Cor. 16.21 and Lemma 15.21 we have 



A M (e, t){f\M)A M (e, t){f\ £l )d[ 9l ]d[k 2 ] 

M/0(q-p)U(p) 

^(p(ffifc 2 )) 2t |(A x (e,0/)(5ifc 2 )| 2 A, t ( 5l; fc 2 )d[ 5l ]d[A :2 ] 

M/0(q-p)U(p) (4) 

<C [ _ \A x (e,t)f(g 1 k 2 )\ 2 A e . t (g 1 ,k 2 )d[g 1 }d[k2] 

JM/0{q-p)U(p) 

=C{Ax{e,t)f,A x {e,t)f) x < oo. 

Therefore, A M (e,t)(f\^) € U. Let V = I°°(e,t)\^. Clearly, V is dense in H. So A M (e,i) 
is a densely defined unbounded operator. It is positive and symmetric by Lemma 17.21 

Now (f,g) e ,t = {A M {e 1 t)f\ K{ ,g\ il ) M for any f,g E I°°(e,t), So C(e,t) = %A M ^,t)- By 
Lemma 2.1, there exists an isometry lA M (e,t) mapping from C(e,t) into %. 

Suppose that lA M (e,t) ^ s not onto. Let / e H such that for any u G V(((Am (e, t)+I)o— -T) 3 ), 

(/,((A M (e,t)+/)o-/)*ttV=0. 

Notice that 

/°°(e,t)| A C X>((^(e,t) +/) - I) C D(((A w (e,t) + J) - I)*), 

and 

(A w (e, t) + i") - I)'(^(£, t) + I)o - I)» = (A M (e, t) + I) - I. 
In particular, 

((A M (e,t)+I) - J)4l°°(e,t)|jfr C P(((^ M (e,t) +/)o - I)*). 

It follows that 



(/,A M (e,t)r»(e,t)| J& )M 
= (/, ((A«(e,t) + /) - 7)/°°(e,t)|^) M 

= (/, ((^mM) + J)o - ^Pm(M)+/)o - r)$r°(e t t)\ A ) M 
=0. 

Let / ej t be a function such that fe,t\M = f ano - 

/ e>t (ff/n) = (// ® ^^(r 1 )/^^) (/ e L, n e N). 



(5) 
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The function / £]t is not necessarily in I(e,t). By Lemma 17.21 V u <E V(e,t), 
= (f,A M (e,t)(u\ M )) M = (fe tU u)e,t = (fe : t,A x (e,t)u) X - 

This equality is to be interpreted as an equality of integrals according to the definitions of 
( , )m and ( , )x- Since Ax{e,t) acts on [/(n)-type in V(e,t) as a scalar, Ax{e,t)V(e,t) = 
V(e,t). We now have 

(f etU V(e,t)) x =0. 

In particular, / c ,t|y( n ) G Therefore = 0. We see that /4 A ,(M) * s an isometry 

from C(e, t) onto 

□ 



The Hilbert space 



is the mixed model for /(e, 0). We now obtain an isometry from C(e, t) onto I(e, 0). Denote 
this isometry by U(e, t). Now, with in the mixed model, the action of I(e, t)(g\k2) is simply 
the left regular action and it is independent of t. We obtain 

Lemma 7.3. Suppose t < 0. Let g e Sp(p, R) or g £ ^(<z)- £et L(<?) be the left regular 
action on 

As a operator on I°°[e,t)\fy, L(g) commutes with AM(e,t). Furthermore, L(g) commutes 
with (A M (e,t) + I) - I. 

Proof: Let g e M. Both Ajv/(e,f) and L(g) are well-defined operator on I°°(e, t)\ M . 
Restricting A(e, t)L(e, t)(g) = I(e, —t)(g)A(e,t) onto M, we have 

A M (e,t)L(g) = L(g)A M (e,t). 

It follows that 

(A M (e,t) + I)L(g) = L(g)(A M (e,t) + 1). 

Recall that (Ajvf(e,i) + -Oo can be defined as the inverse of (AM(e,t) + I) -1 , which exists 
and is bounded. So L(g) commutes with both (Am(£, t) + I)^ 1 and (Am (e, t) + I)q. □ 

Lemma 7.4. We have, for g E M, U(e,t)I(e,t)(g) = I(e,0)(g)U(e,t). 

Proof: It suffices to show that on the mixed model, ((AM(t,t) + I)q — -T) 5 commutes 
with L(g). This follows from Lemma 17.31 □. 

Our main theorem is proved. 



References 

[1] [Bar] V. Bargmann, "Irreducible unitary representations of the Lorentz group " , Annals 
of Math., (Vol 48), 1947 (568-640). 



11 



[2] [BOO] T. Branson, G. Olafsson, B. Orsted, "Spectrum Generating Operators and Inter- 
twining Operators for Representations Induced from a Maximal Parabolic Subgroup" , 
Journal of Functional Analysis, (Vol. 135), 1996, (163-205). 

[3] [He060] Hongyu He, Functions on Symmetric Spaces and Oscillator Representation, 
to appear in Journal of Functional Analysis, |http: / /www.arxiv.org/mat h.RT / 0605595, 
2006 

[4] [He06] Hongyu He, Orbits on Lagrangian Grassmanian, preprint, 
|http://ww.arxiv.org/math.GR/0608306[ 2006. 

[5] [Howe] R. Howe and E.-C. Tan Non-Abelian Harmonic Analysis: Applications of 
SL(2,R), Springer- Verlag, 1992. 

[6] [KR] S. Kudla and S. Rallis, "Degenerate Principal Series and Invariant Distribution", 
Israel Journal of Mathematics, (Vol. 69, No. 1), 1990, (25-45). 

[7] [OZ] B. Orsted, G. Zhang, "Generalized Principal Series Representations and Tube 
Domain ", Duke Math. Journal, (Vol. 78), 1995 (335-357). 

[8] [RS] F. Riesz, B. Sz.-Nagy, Functional Analysis, New York, Ungar 1955. 

[9] [Puk] L. Pukanszky, "The Plancherel formula for the universal covering group of 
SL(R, 2) ". Math. Ann. 156 1964,(96-143). 

[10] [Sahi] S. Sahi, "Unitary Representations on the Shilov Boundary of a Symmetric 
Tube Domain", Representation Theory of Groups and Algebras, (275-286), Comtemp. 
Math. 145, Amer. Math. Soc, Providence, RI, 1993. 



12 



Restrictions of Certain Degenerate Principal Series of the 
Universal Covering of the Symplectic Group 



Hongyu He * 
Department of Mathematics, 
Louisiana State University, 
Baton Rouge, LA 70803, U.S.A. 
email: livingstone@alum.mit.edu 



Abstract 

In this paper, we study the restrictions of degenerate unitary principal series I(e, t) 
of Sp(n,M), the universal covering of the symplectic group, onto Sp(p 7 M.)Sp(n — p, R). 
We prove that if n > 2p, I( e ^)\sp( P M)Sp(n-pR) ^ s un harily equivalent to an i 2 -space of 
a homogeneous line bundle L 2 (Sp(n — p, R) x gX(„„ 2 p)JV ^-M+p) ( see Theorem II We 
further study the restriction of complementary series C(e,t) onto U(n — p)Sp(p, R). We 
prove that this restriction is unitarily equivalent to J(e, t)\fj/ n _ p )sp/ p r) f° r t £ Our 
results suggest that the direct integral decomposition of C(e, t)\g^, R )§^,( n _ p K ) will produce 
certain complementary series for Sp{n — p, R). 

1 Introduction 

Let Sp(n,M.) be the universal covering of Sp(n,R). Sp(n,R) is a central extension of 
Sp(n,R): 

1 ->• C -> Sp(n,R) -> Sp(n,R) -> 1, 

where C = Z. The unitary dual of C is parametrized by a torus T. For each k G T^denote 
the corresponding unitary character of C by x K ■ We say that a representation tt of Sp(n, R) 
is of class /c if n\c — X K ■ Since C commutes with Sp(n, R), for any irreducible representation 
7r of Sp(n,R), n\c = X K f° r some k. 

Denote the projection Sp(n,R) — > Sp(n,M.) by p. For any subgroup H of Sp(n,R), de- 
note the full inverse image p _1 (P) by H. We adopt the notation from [14] . Let P be the 
Siegel parabolic subgroup of Spin, R). One dimensional characters of P can be parametrized 
by (e,t) where e £ T and f 6 C. Let J(e, t) be the representation of Sp(n,R) induced from 
the one dimensional character C £i t parametrized by (e, t) of P. If t € iR and t ^ 0, I(e, t) is 
unitary and irreducible, /(e, i) is called unitary degenerate principal series. If t is real, then 
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I(e, t) has a nontrivial invariant Hermitian form. Sahi gives a classification of all irreducible 
unitarizable I(e,t). If /(e,0) is irreducible, there are complementary series C(e,t) for t 
in a suitable interval ([II])- Some of these complementary series are obtained by Kudla- 
Rallis 9 j, Orsted-Zhang [TT], Branson-Orsted-Olafsson [3J, Lee [TU]. Strictly speaking C(e, t) 
should be called degenerate complementary series because there are complementary series 
associated with the principal series, which should be called complementary series ([5], pQ). 
Throughout this paper, complementary series will mean C(e, t). 

Let [Sp(p, R),Sp(n—p, R)) be a pair of symplectic groups diagonally embedded in Sp(n, R) 
(see Definition [3TTJ). Let U(n) be a maximal compact subgroup such that Sp(n—p,R)CiU(n) 
and Sp(p,R) (~1 i/(ro) are maximal compact subgroups of Spin — p, R) and Sp(p, R) respec- 
tively. Denote 5p(ra -p,R) D f/(n) by ?7(n -p) and Sp(p,R) CI E/(ra) by E/(p). The main 
results of this paper can be stated as follows. 

Theorem 1.1. Suppose p < n~p andt e iR. Let P p , n -2 P be a maximal parabolic subgroup 
of Sp(n — p, R) with Langlands decomposition Sp(p, M.)GL(n — 2p)N Pin -2p- Let A4 e t be the 
homogeneous line bundle 

Sp(n-p,R) ><GL( n ^2p)N p , n - 2p C M+p S'pC" -p,M.)/GL(n - 2p)N p ^ 2p 

(- Sp(p, M)(7(ri - p)/U{p)0(n - 2p)), 

where p = -^j^. £et dgid^] oe an Sp(p, R)J7(n — p) -invariant measure. Then 

I ( e >*)lsS(p,R)s£(»-p,R) - ^(^t.dftdffa]), 

077, which Sp{n — p, R) acis /row. i/ie Ze/t and Sp(p, R) acts /rcw i/ie rig/it. 

Theorem 1.2. LetC(e,t) be a complementary series representation. Suppose thatp < n—p. 
Then 

C(. € >t)\u(n- P )S P {p,M) — ^( £ ' ty\u(n~p)Sp(p,M) ~ ^)'U(n-p)Sp(p,M) E R )" 

p = is the best possible value for such a statement. In particular, for Sp(2m + 1,R) 

^( e '^)lsp(m+l,R) ^ ^( £ '*)ls^(m+l,R)- 

To see this, let L 2 (Sp(n, R)) K be the set of functions with 

f(*g) = x K {z)f{g) (» £C,ge Sp(n, R)); 
ll/f - / \f(9)\ 2 d[g] < oo ( s G Sp(n,R), [g] G Sp(n,R)). 

JSp(n,U) 

We say that a representation of class k is tempered if it is weakly contained in L 2 (Sp(n, R)) K . 
By studying the leading exponents of I(e, 0) and C(e, t), it can be shown that J(e, 0)|g^ m+1 R ) 
is "tempered " and C(e, Ols'ftn+i R) * s n0 ^ "t em P ere d"- Therefore 

/ ( £ '°)ls^(m+l,R) ^ C '( e '*)ls^(m+1,R)- 

The author would like to thank Professors G. Olafsson and J. Lawson for very helpful 
discussions. 
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2 A Lemma on Friedrichs Extension 

Let S be a semibounded densely denned symmetric operator on a Hilbert space H. S is 
said to be positive if (Su,u) > for every nonzero u e T>{S). Suppose that S is positive. 
For u, v e P(S), define 

(u,i>)s = (u,Sv), 
\\u\\ 2 s = (u,Su). 

Let ffg be the completion of T>(S) under the norm || Clearly Hs+i Q H and H$+i Q 
Hs- 

The operator S + I has a unique self-adjoint extension (S + I)o in H , the Friedrichs exten- 
sion. (S + I)q has the following properties 

• V(S) C X>((S + J) ) C H s+ i C if; 

• (u, i>)s+i = (u, (S" + 7)o«) for all u G H s +i and u e + J) ) 

(see Theorem in Page 335 |12]). Now consider (5 + /)o — /. It is an self-adjoint extension 
of S. It is nonnegative. By the spectral decomposition and functional calculus, (S + J)o — / 
has a unique square root T (See Pg. 127. 128. [H]). 

Lemma 2.1. Let S be a positive densely defined symmetric operator. Then the square root 
of (S + I)o — I extends to an isometry from H$ into H . 

Proof: Clearly, the spectrum of T is contained in the nonnegative part of the real line. 
By spectral decomposition T>((S + I) - I) = V((S + J) ) C V(T) and TT = (S + I) - I. 
In addition for any u, v G V(S) C D((S + I) ), 

(Tu, Tv) — [u, TTv) — (u. (S + I)qv — v) = (u, Sv) — (u, v)s- 

So T is an isometry from TJ(S) into H. Since T>(S) is dense in Hs, T extends to an isometry 
from Hs into H. □ 

3 Degenerate Principal Series of Sp(n,M) 

Fix the Lie algebra: 

ap(n,R) = {( X z Y xt ) \ Y*=Y,Z t = Z} 
and the Siegel parabolic algebra: 

Fix the Levi decomposition p = I © n with 

( = {(o °_ x t ) \x&Qi(n,M)}, tt = {(o ^)|y* = y>. 

Fix a Cartan subalgebra 

o = {diag(Fi, ff 2 , . . . , ff n , -Fi, -H 2 ..., -H n ) \ H e M}. 
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Let Sp(n, R) be the symplectic group and P be the Siegel parabolic subgroup. Set U (n) = 
Sp(n,R) fl 0(2n) where 0(2n) is the standard orthogonal group. Let LN be the Levi 
decomposition of P and A be the analytic group generated by the Lie algebra a. Clearly, 
L = GL(n, R) and L fl U(n) = 0(n). On the covering group, we have L n U(n) = 0(n). 
Recall that 

U(n) — {(x,g) | g G U(n), exp 2nix = detg,x £ R}. 

Therefore 

O(n) = {(a;, 5) | g G 0(n), exp 27ria; = det(/,x G R}. 
Notice that for g G O(n), dct g = ±1 and x G \ r L. We have the following exact sequence 

1 -»■ SO(n) 6{n) -> -Z -> 1. 

Consequently, we have 

1 -> GL (n, M) -> £ -> ^Z ->■ 1. 

In fact, 

L = Ux, q) I a E L, exp 27rix = . - —r .X G K}. 
lv ; 1 |detg| 

The one dimensional unitary characters of are parametrized by the one dimensional 
torus T. Identify T with [0,1). Let /i e be the character of |Z corresponding to e G [0,1) 
Now each character /i £ yields a character of L, which in turn, yields a character of P. For 
simplicity, we retain // to denote the character on L and P. Let ^ be the det-character on 
L , i.e., 

v(x,g) = |detfl| {x,g)eL. (2) 
Let _ 

be the normalized induced representation with e G [0, 1) and t G C. This is Sahi's notation 
in the case of the universal covering of the symplectic group ([14 ). I(e,t) is a degenerate 
principal series representation. Clearly, J(e, t) is unitary when t G iR. 

When i is real and I(e, t) is unitarizable, the unitary representation, often denoted by C(e, t), 
is called a complementary series representation. Various complementary series of Sp(n, R) 
and its metaplectic covering was determined explicitly or implicitly by Kudla-Rallis, Orsted- 
Zhang, Brason-Olafsson-Orsted and others. See [9], [3], [11] and the references therein. The 
complete classification of the complementary series of the universal covering is due to Sahi. 

Theorem 3.1 (Thm A, |14j). Suppose that t is real. For n even, I(e,t) is irreducible and 
unitarizable if and only if0< \t\ < \i — \2e — l\\. For n odd and n > 1, I(e,t) is irreducible 
and unitarizable if and only if < \t\ < ^ — \h — |2e — 1||. 

One can easily check that the complementary series exist if e 7^ 0, | for n odd and n > 1 ; 
if e 7^ a ,| for n even. It is interesting to note that complementary series always exist unless 
I(e, t) descends into a representation of the metaplectic group. For the metaplectic group 
Mp(2n+1, R), there are two complementary series t)(0 < t < |) and i"(|,£)(0 < t < i). 
For the metaplectic group Mp(2n, R), there are two complementary series 7(0, £)(0 < £ < |) 
and l(^,t)(0 <t<i). These four complementary series are the "longest". 
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For n = 1, the situation is quite different. The difference was pointed out in [5]. For 
example, there are Bargmann's complementary series representation for 1(0, t) (t G (0, i)). 
The classification of the complementary series of 5p(l,K) can be found in [2J, [T3], [7]. 

Since our restriction theorem only makes sense for n > 2,^we will assume n > 2 from 
now on. The parameters for the complementary series of Sp(n, R) are illustrated in fig. 

EI 



4 The generalized compact model and The Intertwining 
Operator 

Recall that 

I°°(e, t) = {f G C™(Sp(n, R)) | /(^n) = (m^^X/" 1 )/^, G? G Stfn, R), I G L, n G N)} 

where p = Let X — Sp(n,M)/P. Then I°°(e,t) consists of smooth sections of the 

homogeneous line bundle C £t t 

Sp(n,R) Xp C^Qyt+p -> X. 

Since X = U(n)/0(n), U(n) acts transitively on X. The function / G I°°(e,t) is uniquely 
determined by f\fj/ n \ and vice versa. Moreover, the homogeneous vector bundle £ e ,t ca n be 
identified with /C e ,t 

C/ ( n ) X 6(r>) C M e ®f t+f, l6(n) ^ ^ 

naturally. Notice that the homogeneous line bundle ]C e> t does not depend on the parameter 
t. We denote this line bundle by JC e . The representation I°°(e,t) can then be modeled on 
smooth sections of /C e . This model will be called the generalized compact model. 

Let d[k] be the normalized J7(n)-invariant measure on X. The generalized compact model 
equips the smooth sections of /C £i t with a natural pre-Hilbert structure 

[h,h)x= I h(k)h(k)d[k}, 

J[k]ex 

where k G U(n) and [k] E X. It is easy to verify that /i(fc)/2(&) is a function of [k] and it 
does not depend on any particular choice of k. Notice that our situation is different from 
the compact model since U(n) is not compact. We denote the completion of I°° with respect 
to (,) x by Ix{e,t). 

Secondly, the action of U(n) on IC e induces an orthogonal decomposition of Ix(e,t): 

Ix(e, t) = ® ae2Z ,V(a + e(2, 2, . . . , 2)), 

where V(a + e(2, 2, . . . , 2)) is an irreducible finite dimensional representation of U(n) with 
highest weight a + e(2, 2, . . . , 2) and a satisfies 

ai > a 2 > ■ ■ ■ > a n . 
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This is essentially a consequence of Helgason's theorem. Let 

V(e,t) = ® ae2Z nV(a + e(2,2,...,2)). 

V(e, t) possesses an action of the Lie algebra sp(n,R). It is the Harish-Chandra module of 
I(e,t). Clearly, V(e,t) C I°°(e,t) C I x (e,t). 

For each t, there is an Sp(n, R)-invariant sesquilinear pairing of Ix(e,t) and Ix(e,—i), 
namely, 

Cfi,/ 2 )= / h(k)h(kjd[k}, 

where /i G Ix(c,t) and /2 G I X ( e i —t). If i G iR, we obtain a Sp(n, R)-invariant Hermitian 
form which is exactly ( , )x- Since ( , )x is positive definite, Ix(c,t) is a unitary represen- 
tation of Sp(n, E). 

For each real t, the form ( , ) gives an sp{n, R)-invariant sesquilinear pairing of V(e,t) and 
V(e, —t). In addition, there is an intertwining operator 

A(e,t):V(e,t)^V(e,-t) 

which preserves the action of sp(n,R) (see for example [3]). Define a Hermitian structure 
(, ) e , t onV(e,t) by 

(it, v) e , t = (A(e, t)u, v), (u, v G V(e, t)). 

Clearly, ( , ) e t is sp(n, R)-invariant. So A(e,t) induces an invariant Hermitian form on 
V(e,t). 

Now A(e,t) can also be realized as an unbounded operator on I x (^,t) as follows. For 
each / G V(e,t), define A x (e,t)f to be the unique section of £ £i t such that 

(A x (€,t)f)\ &(n) = (A(e,t)f)\ &{n) . 

Notice that A x (e,t)f G I(e,t) and A(e,t)f G /(e, -t). They differ by a multiplier. 

Now A x (e,t) is an unbounded operator on the Hilbert space Ix(^,t). The following fact is 
well-known in many different forms. I state it in a way that is convenient for later use. 

Lemma 4.1. Lett G M. I(e,t) is unitarizable if and only ifA x (c,t) extends to a self-adjoint 
operator on Ix(e,t) with spectrum on the nonnegative part of the real axis. 

The spectrum of Ax(e,t) was computed in [3] and [11] explicitly for special cases and 
in |14j implicitly. In particular, Ax{t,t) restricted onto each £/(n)-type is a scalar multipli- 
cation and the scalar is bounded by a polynomial on the highest weight. We obtain 

Lemma 4.2 (15:). Ax(e,t) extends to an unbounded operator from 7°°(e, t) to I°°(e,t). 

This lemma follows from a standard argument that the norm of each [/(n)-component 
in the Peter- Weyl expansion of any smooth section of /C e decays rapidly with respect to the 
highest weight. It is true in general (see [15]). 
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5 Actions of Sp(p, R)Sp(q, R) 



Suppose that p + q = n and p < q. Fix a standard basis 

{ei,e 2 ,...,e p ;e 1 ,e2,...e*} 
for the symplectic form fi p on R 2p . Fix a standard basis 

\Jli J2> • • • , Jqi J 1 1 J 2 i ■ • ■ i Jq 5 

for the symplect form Vl q on M. 2q . 

Definition 5.1. Let Sp(p,M) be the symplectic group preserving f2 p and Sp(q,M) be the 
symplectic group preserving Q q . Let 

and Sp(n,M) be the symplectic group preserving Q. We say that {Sp(p, R), Sp(q, R)) is 
diagonally embedded in Sp(n,W). 

We shall make a remark here. In [6 , f2 = il p + £l q . Sp(p, R)S , p(g, R) is embedded differently 
there. The effect of this difference is an involution r on the representation level. 

Let Pp,q~ P be the subgroup of Sp(q,M) that preserves the linear span of {/ p +i, . . . , f q }. 
Choose the Levi factor GL(q — p)Sp(p,R) to be the subgroup of Pp, q - P that preserves the 
span of {/p+i, ■ • ■ , f q }- In particular the Sp(p, R) factor can be identified with the symplec- 
tic group of 

span{/i,...,/ p ; /*,.../*}, 

which will be identified with the standard Sp(p, R). More precisely, for x G Sp(p, R), by 
identify e, with fa and e* with /* and extending x trivially on f p +i, . . . f q ; /p+i, ■ • ■ , f q , we 
obtain the identification 

x £ Sp{p, R) — !> ± £ Sp(q, R). (3) 
Now fix a Lagrangian Grassmanian 

s = span{ei + f 1 ,...,e p + f p , e* +/*,..., e* + fp,f P +i, ■ ■ ■ fq}- 

Then the stabilizer Sp(q, R) xo = GL{q — p)N p ^ q - p where N Pl q— p is the nilradical of P Piq - p . 
Put 

A(Sp(p,R)) = {(«,«) | u G SpfeR)} C 5p(p,R)S*p(g,R)} 

and 

H = A(Sp(p, R))GL(q - p)Np^ p . 

Lemma 5.1 ( 6 ). Let p < q and p + q = n. Let Xq be the Sp(p,M.) x Sp(q,M) -orbit 
generated by xq. Then Xq is open and dense in X and [Sp(p,M)Sp(q,M)] Xo = H. 

Notice here that Xo depends on (p,q). Let P = Sp(n,M.) X(j . The smooth representation 
I°°(e,t) consists of smooth sections of L tt t ■ 

Sp(n, R) Xp C^^t+p -» X. 

Consider thejubgroup Sp(p,R)Sp(q,R) in Sp(n,R). Notice that Sp(p,R) l~l Sp(q,M) = Z. 
So Sp(p,W)Sp(q,M.) is not a direct product, but rather the product of the two groups as 
sets. 
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Definition 5.2. For any f £ Ix(e,t), define 

fx a = f\sp( p M)Sp(q,M)- 

Let L^° Xo (^e,t) be the set of smooth sections of C t j that are compactly supported in Xq. 
Clearly fx is a smooth section of 

Ship, ®)Sp{q, R) C p . 0tf t+p -> X . 

Notice that A(Sp(p,R)) sits inside of SL(n,M) C GL(n,R) C P. The universal covering of 
Sp(n,R) splits over SL(n, R) C P. Similarly the universal covering of Sp(q,M) also splits 
over Np^ q -p. So we have 

H = A(S P (p, R))GL{q - p)N p<q - p , 

where GL(q — p)N p>q - p C Sp(q,M). In particular, /i e ® & / ' +p |a(Sp(p,r))./v p is trivial and 
(gi i y *~ l_p jd) * s essentially the restriction from GL(p + q) to GL(q — p). If p = q, then 
GL(0) will be the identity element. So GL(0) is just C. We have 
Lemma 5.2. TTie identification 

.x e SpO?, K) zSpfa, R) 

Zafts natually to Sp(p,R) -> £p(g,R). Let e P°(e,t). Then 

0(5i,52) = 0(l,525i" 1 ) (51 6 Sp(p,«),fl2 G Sp(<7,R)). 

/n addition 

4>(l,g 2 h) = // ® ^"(/i-^^Cl,^) (/i € GX( g -p)N p ^p). 

Now let us consider the action of Sp(p,M.) and Sp(q,M) on I(e,t). By Lemma f5.2[ we 
obtain 

Lemma 5.3. Let 4> £ I°°(e,t) and hi £ Sp(p,R) and g 2 £ Sp(q,R) . Then 

[I(e,t){h 1 )4>](l,g 2 ) = f(l,g 2 tii)- 
In particular the restriction map 

4, £ r»(e,t) H- % (g , R) G C°°(Sp(,,R) x^^^ C p . a ^) 

intertwines the left regular action of Sp(p,M) on I°°(e,t) with the right regular action of 
Sp(p,R) on C°°(5p(q,R) ^ai( q - P )N p , q . p 

Obviously, the restriction map also intertwines the left regular actions of Sp(q,M). 



8 



6 Mixed Model 

Now fix complex structures on R 2p and R 29 and inner products ( , ) p , ( , ) q such that 

= ^( I )pj = ~ 3f( ) )q- 

Let ?7(p) and {/ (q) be the unitary groups preserving ( , ) p and ( , ) q respectively. U (p) and 
U{q) are maximal compact subgroups of Sp(p, R) and Sp(q, R). Let ?7(n) be the unitary 
group preserving ( , ) p + ( , ) q . Then ?7(n) is a maximal compact subgroup of Sp(n,R). In 
addition, 

E7(p) = S P (p, R) n f/(n) [/(?) = Sp(q, R) n f/(n). 

Identify U(q)C\P p , q - p with 0{q-p)U{p). Recall that X = Sp{q,R)/GL(q-p)N p . q - p . The 
group >Sp(p, R) acts on freely from the right. We obtain a principal fibration 

Sp(p,R) -»■ X -»■ Sp(o,R)/P p , 9 _ p - U{q)/0(q-p)U(p). 

Let c?c/i be a Haar measure on Sp(p, R) and o^fe] be an invariant probability measure on 
U(q)/ 0(q — p)U(p). Then dg\d[k 2 ] defines an U(q)Sp(p, R) invariant measure on Xo- 

Definition 6.1. Let M = Sp(p,R)U(q) C R)S'p(g, R) C Sp(n,R). Elements in 

X are parametrized by a pair (gi, [k 2 ]) /or (gi,k 2 ) G M. for eac/i <? 6 5p(n,R), write 
.9 = u(g)p(g) where u(g) 6 f/(n) and p(g) 6 P ; ^ e identity component of P. For each 
{gi,k 2 ) e (5p(p,lR), we ftawe 

51*2 = u{gik 2 )p{gik2) = k 2 u{g 1 )p(g 1 ). 

The component u defines a map from M to U(n). In particular, u induces a map from 
M/0(q — p)U(p) to U(n)/0(n) which will be denoted by j. The map j parametrizes the 
open dense subset Xq in X by 

e Sp(p,R)/Cx U{q)/d{q-p)U{p). 

Change the variables on X from M/0(q — p)U{p) to U(n)/0{n). Let J([.9i], [k 2 ]) be the 
Jacobian: 

d[gi]d[k 2 ] ' 

J can be regarded as a function on Sp(p,R)U(q) or Sp(p,R)U(q)/U(p)0(q—p), even though 
it is defined as a function on the covering. Denote the line bundle 

Sp(q,R) *GL {q _ p)Npq _ p C^^ t +P X Q . 

by M. e .t- Denote the line bundle 

M X6 {g - P )u( P ) *V M/0(q- P )U(p) = X . 

by M t . 

Clearly I^° Xo ( € ^) c Consider the restriction of ( , ) x onto I^° Xo (e,t). We are 

interested in expressing (,)x as an integral on M/0(q — p)U(p). This boils down to a 
change of variables from U{n)/0(n) to M/Q(q ~ p)U(p). We have 
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Lemma 6.1. Let A t (gi,k 2 ) = v(p(g 1 )) t+t+2p J{[gi], [k 2 ]) (see Equ.ffl)). Then for every 
f\ifi E I°°(e,t) we have 



{h,h)x = / h{9ik 2 )f 2 {gik 2 )A t {gi,k 2 )d[ gi ]d[k 2 } 

JM/6(q-p)U(p) 

where g\ E Sp(p, M), k 2 E £/(g), [51] € <5p(p, K) and [k 2 ] E U(q)/0(q—p)U(p). Furthermore, 
At(gi,k 2 ) is a nonnegative right 0(q — p)U(p) -invariant function on M. 



Proof: We compute 

/ fi (gi k 2 ) f 2 (gi k 2 ) A t (gi , k 2 )d[g 1 ]d{k 2 

J M /0(a—v)U(v) 



M/0(q-p)U{p) 

/i("(5ife))/ 2 (u(ffife))^(p(gi))- t ^ 2p A t ( 5l ,fc 2 )d[ 5l ]rf[A :2 ] 

M/0(q-p)U(p) 



/i(u(ffiA ;2 ))/ 2 (u( ffl A ;2 ))Kp(gi))-™ p A t ( 5l ,fc 2 )J- 1 ( 5l ,fc 2 HX[3i],[fc 2 ]) 

M/6(q-p)U(p) 



fi(u)f 2 (u)d[u] = (fi,f 2 )x. 

Xo 

(4) 

Since v{p{gi) and J([<?i], [k 2 ]) remain the same when we multiply k 2 on the right by 
0(q —p)U{p), A t {gi,k 2 ) is a nonnegative right 0(q — p)[/(p)-invariant function. □ 

Combining with Lemma l5.3[ we obtain 

Corollary 6.1. As representations of Sp(p,M)Sp(q,R) , 

I x (e,t)^L 2 (M £ , t ,A t d[gi]d[k 2 ]). 

For each f x ,f 2 E /~ Xo (e,t), define 

(A,/2)M,t= / /1 (9i fe) A (<?i ) A t (31 A; 2 )d[<n Mfo] , 

JM/6(q-p)U(p) 



(fi,h)M= / /i(gi,fc2)/2(.9iA: 2 )d[5i]d[fc 2 ]. 

JM/d(q-p)U(p) 

The completion of (e, t) under ( , ) M , t is i 2 (X e ,t, A t d[ 5l ](i[fe 2 ]). Wecall L 2 (A^AtdMd[/c 2 ]), 

the mixed model. We denote it by Im{c, t)- On (e, , the actions of Sp(p, K) andSp(g,R) 
are easy to manipulate. 

7 Mixed Model for Unitary Principal Series 

Lemma 7.1. IftG iR, then At(gi,k 2 ) is a constant and (, )hi.t is a constant multiple of 
( , )m- 
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Proof: Let t £ iM.. Let /i, /a £ I°°(^, t) and h £ Sp(p, K). Recall that Xo is parametrized 
by a pair [gi] £ Sp(p,R)/C and [k 2 ] S U(q)/0(q —p)U(p). By Lemma I5TT1 we have 



(I{e,t){h)h,I{e,i){h)h)x 

/ 1 (/ l - 1 5l A :2 )/ 2 (/ 1 ~i ffl fc 2 )A t ( ffl ,fc 2 )d[ 5l ]d[fc 2 ] 



A'n 



(5) 



/i (51 ) h (51 , fa ) A t (hgi , fa ) d\gi] d[k 2 ] 
Since I(e,i) is unitary, (I(e,t)(h)fi,I(e,t)(h)f 2 )x = {h,h)x- We have 



fi{gik 2 )f 2 {gik 2 )A t {hg 1 ,k 2 )d[gi}d[k 2 } = / h{9ik 2 )f 2 {gik 2 )A t {gi, k 2 )d[gi]d[k 2 ]. 
Xq J Xo 

It follows that At(hgi, k 2 ) — A t (<?i, A: 2 ) for any ft, € Sp(p, M). Similarly, we obtain A t (<?i, kk 2 ) 
A(<7i, k 2 ) for any k £ U(q). Hence, A t (gi, k 2 ) is a constant for purely imaginary t. □ 

Combined with Cor. 16.11 we obtain 



Theorem 7.1. Let t £ iM.. The restriction map f — > fx a induces an isometry be- 
tween I(e,t) and L 2 (A^ Cj t, rf[<7i]d[fa]). In addition, this isometry intertwines the actions 
of Sp(p,M.)Sp(q,M). So as Sp(p,M.)Sp(q,M) representations, 

I(e,t) Si L 2 {M t , u d[ 9l ]d[k 2 ])- 
and as Sp(p,M.)U(q) representations, 

I(e,t) = L 2 (M^d[ gi }d[k 2 })- 

Notice thatL 2 (A^ e , d[<7i]<i[fa]) does not depend on the parameter t. The following corol- 
lary is automatical. 

Corollary 7.1. Suppose that p + q = n and p < q. For t real, 

/ ( e '^)ls P ( P ,R)[/(g) " / ( e '°)ls P ( P ,R)C/(g) ~ L 2 (M e , d[ 9l }d[k 2 }) . 

For t a nonzero real number, A t (<?, fc) is not a constant. So C(e,t) cannot be modeled 
naturally on L 2 (M. t tl d[gi]d[k 2 ]). Nevertheless, we have 

Theorem 7.2 (Main Theorem). Suppose thatp + q = n andp < q. Given a complementary 
series representation C(e,t), 

C ( e ^)lsp(p,R)£/( 9 ) - / ( £ ' )lsp( P ,K)[/(9) =L 2 {M t ,d[ gi ]d[k 2 ]). 

In other words, there is an isometry between C{e,t) and I{e, 0) that intertwines the actions 
ofU{q) and ofSp(p,R). 

We shall postpone the proof of this theorem to the next section. We will first derive 
some corollaries from Lemma 17. II concerning A and v{gi). 

Corollary 7.2. J([<7i], [k 2 ]) — cv{p{g\))~ 2p for a constant c and A t (gi, k 2 ) = cv(p{gi)) t+t . 
So both A t and J([gi], [fa]) do not depend on k 2 . Furthermore, 

I(e,t)~L 2 (M e , t ,v( P (g 1 )) t +~ t d[g 1 }d[k 2 })=I M (e,t). (6) 
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v(p(gi)) is a function on Sp(p,M,)/C. So it can be regarded as a function on Sp(p,R). 
Corollary 7.3. v(p(gi))~ p G L 2 (Sp(p, R)) and v{p{g\))^ 1 is a bounded positive function. 
Proof: Since X is compact, 

I v( P ( 9l )r 2p d gi = C I J([ 9l ], [k 2 ])d[ 9 t]d[k 2 ] = C [ ld[k] < oo. 

JSp(p,M) J M/6(q-p)il(p) JU(n)/d(n) 

So v(p(gi))~ p G L 2 (Sp(p, R)). Now we need to compute v(gi). Recall that P is defined to 
be the stabilizer of 

x = span{ei + ft, e p + f p , e* + /*,..., e* + /*, f P +i, ■ ■ ■ f q }. 

So j(gt, 1) is the following Lagrangian 

span{#iei + / 1; . . .,g x e p + f p , 9 te\ + ft,-- -,gie* + f*,f p+ x, . ../,}• 

The action of U(n) will not change the volume of the n-dimcnsional cube spanned by the 
basis above. So v(p(gi)), as the determinant character, is equal to the volume of the n- 
dimensional cube, up to a constant. Hence 

KK<7i)) = [2- n det(< ?1 ^+/)]i 
Clearly, ^ip(gi))^ 1 is bounded and positive. □ 

This corollary is easy to understand in terms of compactification. Notice that the map 
j, without the covering, 

Sp(p,R)U(q)/U(p)0(q-p) U(n)/0(n) 

is an analytic compactification. Hence the Jacobian J(gt, [k 2 ]) should be positive and 
bounded above. Since J(gi, [k 2 ]) = cv{p(gi))~ 2p , ^(pigi))^ 1 must also be positive and 
bounded above. The situation here is similar to [4] (see Appendix) and [5] (Theorem 2.3). 
It is not clear that j(gi, 1) gets mapped onto U(2p)/0(2p) though. 

If / G /A/(e,ti) and h > 0, by Cor. 17.31 and Equation 0, we have \\f\\M,t t -h < C||/||M,ti- 
So Jm(Mi) C hi(t,ti - h). 

Corollary 7.4. Suppose that h > 0. Then Im^^i) C Im^^i — h). 

8 "Square Root "of the Intertwining Operator 

Suppose from now on t G R. For / G I°°(e, t)]^, define a function on M, 

(A M (e,t)f)(gtk 2 ) = A{e,t)f{g x k 2 ) ( 9l G Sp{p,R),k 2 G U(q)). 

So Am(c, t) is the "restriction "of A(e, t) onto M. Am(c, t) is not yet an unbounded operator 
on Jjvf(e,t). In fact, for t > 0, AM(e,t) does not behave well and it is not clear whether 
j4m(c,^) can be realized as an unbounded operator on lM{c,t). The function AM(e,t)f 
differs from Ax(e,t)f. 
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Lemma 8.1. For t eM and f £ I°°(e,t), 

(A M (e,t)f\^)( 9l k 2 ) = {A x {e,t)f){ 9l k 2 ) v {p{ 9l )) 2t = (A x (e,t)f)( 9l k 2 )A t ( 9l ,k 2 ). 
This Lemma is due to the fact that A x (e,t)f £ -f(e,£) but A(e,t)f £ I(e, —t). 

Let / £ I°°(e,t). In terms of the mixed model, the invariant Hermitian form ( , ) Cj t can be 
written as follows: 

(fJ)e,t = (A x (e,t)f,f) x = f A M (e,t)f\ & f\^d[ gi }d[k 2 ]. 

JM/0(q-p)U(p) 



This follows from Lemma 18.11 and Lemma 16.11 We obtain 

Lemma 8.2. For h, f 2 £ I°°(e,t), (/i,/ 2 ) £ ,t = {A M {e, t)ft\ a , Mm)m- 

Theorem 8.1. Ift<0 and C(e, t) is a complementary series representation, then Am{z, t) 
is a positive and densely defined symmetric operator. Its self- adjoint- extension (Am^^) + 
I)o — I has a unique square root which extends to an isometry from C{e,t) onto 

L 2 (M e ,d[ 9l ]d[k 2 }). 

Proof: Let t < 0. Put 

H = L 2 {M e ,d[ 9l ]d[k 2 }). 

Let / £ I°°(e,t). Then A M {e,t)(f\^)( gi k 2 ) = v{p{ 9l )) 2t A x {e,t)f{ 9l k 2 ). By LemmaO 
Cor. 17.31 and Lemma[B~Tl we have 



(7) 



A M (e,t)(f\M) A M(e,t)(f\M)d[gi}d[k 2 ] 

M/5{q-p)U(p) 

^( J5 ( 5l )) 2t |(A x (e,t)/)( 5l fc 2 )| 2 Kp(gi)) 2t d[5i]rf[fc2] 

M/0(q-p)U(p) 

iy(p( 9l )) 2t \(A x (e,t)f)( 9l k 2 )\ 2 A t ( 9l ,k 2 )d[ 9 Mk2} 

M/0(q-p)U(p) 

<C [ _ \A x (e,t)f( 9l k 2 )\ 2 A t ( 9l ,k 2 )d[ gi ]d[k 2 ] 

JM/0(q-p)U(p) 

=C(A x (e,t)f,A x (e,t)f) x < oo. 

Therefore, A m (M)(/Im) £ U. Let P = I°°(e,t)\ A . Clearly, X> is dense in H. So A M (e,i) 
is a densely defined unbounded operator. It is positive and symmetric by Lemma l8.2l 

Definition 8.1. Define U{e,t) = ((A M (e,t) + I) - I)?. 

Now (/ )5 ) £ii = {A M (e,t)f\ & ,g\ a ) M for any /,g £ I°°{e,t). So C(e,i) = W^( e , t ). By 
Lemma 2.1, U(e,t), mapping from C(e,t) into H, is an isometry. 

Suppose that U(e,t) is not onto. Let / £ H such that for any u £ V(U(e,t)), 

(f,U(e,t)u) M = 0. 

Notice that 

r( e ,t)l A cpp M (M) + /)o-/)cJ)(M(6,()), 
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and 

U(e,t)U(e,t) = (A M (e,t) + I) -I. 

In particular, 

W( £ ,r(M)lMCP(W(M)). 

It follows that 

(f,A M (e,t)I°°(e,t)\^)M 
= (/, ((A M (e,t) + I) - r)F°(e,t)\x)M , 
= (f,U(e,t)U(e,t)L^(e,t)\^) M [ } 

=0. 

Let / £j t be a function such that fe,t\hi = / and 

h, t {gln) = ® S+'Xr^fM (I e L, » G JV). 

By LemmaEl Vu£ V(e,t), 

= (/, A M (e,<)(u| A : f )) M = A x (e, i)u)x = (/e,t,«)e,t- 

This equality is to be interpreted as an equality of integrals according to the definitions of 
( , )m and { , )x- Since Ax(e, i) acts on [/(n)-types in V(e,t) as scalars, Ax(e,t)V(e,t) — 
V(e,t). We now have 

(f €tt ,V(e,t)) x = 0. 

In particular, fe,t\f/r n ) G L X (X). By Peter- Weyl Theorem, / ejt = 0. We see that U(e,t) is 
an isometry from C(e,t) onto L 2 (.M e , d[gi]d[fe2]) ■ □ 

The Hilbert space L 2 (Ai e , d[gi]d[k2\) is the mixed model for J(e, 0) restricted to M. We 
now obtain an isometry from C(e,t) onto 7(e,0). Within the mixed model, the action of 
I(e,t){g\k2) is simply the left regular action and it is independent of t. We obtain 

Lemma 8.3. Suppose t < 0. Let g G U(q). Let L{g) be the left regular action on 
L 2 (A4 e , d[gi]d[k2\). As an operator on I°°(e, t)\j>, L(g) commutes with A^j(e,t). Further- 
more, L{g) commutes with (Aj\f(e,t) + 7)q — L. Similar statement holds for g G Sp(p,M). 

Proof: Let g G M. Both i) and L(g) are well-defined operator on I°°(e, t)\^. Regard- 

ing A(e, t)I(e, t)(g) = I(e, —t)(g)A(e, t) as operators on the mixed model L 2 {M t , d[gi]d[k 2 }), 
we have 

A M (e,t)L(g) = L{g)A M (e,t). 

It follows that 

L(g)-\AM(e,t) + L)L(g) = (A M (e,t) + I). 

Since L(g) is unitary, L(g)- l {A M {e, t) + I) L(g) = (A M (e, t) + J) Q . In fact, (A M (e, *) + i")o 
can be defined as the inverse of (j4jw(e,t) + i} -1 , which exists and is bounded. So L{g) 
commutes with both (ijkffe*) + and (Ajif(e,t) + 7)o- □ 

Lemma 8.4. We have, for g G M, U(e,t)L(e,t){g) = L(e,0)(g)U(e,t). 
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Proof: Recall from Theorem 17.1 1 that the action of M on the mixed model is independent of 
t. It suffices to show that on the mixed model, U{e,t) commutes with L(g) for any g e M. 
By Lemma [573l 

L{g)- x [(A M {e, t) + I) - I]L(g) = (A M (e, t) + I) - I. 

Since L(g) is unitary on L 2 (M e , d[gi]d[k 2 ]), both sides are positive self-adjoint operators. 
Taking square roots, we obtain L(g)^ 1 U(e,t)L(g) — U(e,t). □. 

Theorem 17. II is proved. 
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